• 专利附录
  • 专利说明
  • 权利要求
  • 类似技术
  • 同族专利
  • 引用文献
  • 法律状态
  • 优先权
    • 专利摘要:
    method and apparatus for using bi-directional (bio) optical flow for a true bi-directional predicted block are disclosed. according to a method of the present invention, a non-split bio process is disclosed, where the movement displacement x and movement displacement y are derived using operations including displacement to the right without any division operation. according to another method, a refined predictor is generated for the current block, applying the bio process to the reference blocks, in which the application of the bio process comprises the application of a bio process dependent on boundary condition conditionally to the associated boundary pixels reference blocks.
    公开号:BR112019018922A2
    申请号:R112019018922
    申请日:2018-03-02
    公开日:2020-04-14
    发明作者:Chen Ching-Yeh；Chuang Tzu-Der；Huang Yu-Wen
    申请人:Mediatek Inc；
    IPC主号:
    专利说明:

    METHOD AND MOVEMENT REFINING APPARATUS BASED ON BI — DIRECTIONAL FLOW FOR VIDEO ENCODING
    CROSS REFERENCE FOR RELATED APPLICATIONS [001] The present invention claims priority for US Provisional Patent Application No. 62 / 472,002, filed on March 16, 2017, US Provisional Patent Application No. 62 / 474,702, filed on March 22, 2017 2017 and US Provisional Patent Application No. 62 / 480,359, filed on April 1, 2017. Provisional US Patent Applications are hereby incorporated by reference in their entirety.
    FIELD OF THE INVENTION [002] The present invention relates to motion compensation using bidirectional optical flow (BIO) to refine movement to a predicted bidirectional block. In particular, the present invention relates to reducing the bandwidth and reducing the computational complexity associated with the BIO process.
    BACKGROUND OF THE INVENTION
    Bi-directional optical flow (BIO) [003] Bi-directional optical flow (BIO) is a motion estimation / compensation technique revealed in JCTVCC204 (E. Alshina, et al., Bi-directional optical flow, Joint Collaborative Video Coding Team (JCT -VC) ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 3 meeting: Guangzhou CN 7 to 15 october 2010, item: JCTVCC204) and VCEG-AZ05 (E .. Alshina, et al performing research tools known for the next generation video coding, ITU-T SG 16 Question 6 video coding Experts Group (VCEG) 52 meeting: 19-26 June
    Petition 870190090540, of 12/09/2019, p. 40/91
    2/35
    2015, Warsaw, Poland, Document: VCEG-AZ05) BIO derived the refinement of sample-level motion based on the assumptions of optical flow and steady motion as shown in Figure 1, where a current pixel 122 in a sliceB (slice of bi-prediction) 120 is predicted by a pixel in the reference image 0 (130) and a pixel in the reference image 1 (110). As shown in Figure 1, the current pixel 122 is predicted by pixel B (112) in reference image 1 (110) and pixel A (132) in reference image 0 (130). In Figure 1, v x and v y are pixel displacement vectors in the x direction and y direction, which are derived using a bidirectional optical flow method (BIO). It is only applied for truly bidirectional predicted blocks, which are predicted from two reference images corresponding to the previous image and the last image. In VCEG-AZ05, the BIO uses a 5x5 window to derive the movement refinement of each sample. Therefore, for an NxN block, the results compensated by movement and the corresponding gradient information of a block (N + 4) x (N + 4) are necessary to derive the sample-based movement refinement for the NxN block. According to VCEG-AZ05, a 6-lead gradient filter and a 6-lead interpellation filter are used to generate the gradient information for BIO. Therefore, the computational complexity of BIO is much greater than that of traditional bidirectional prediction. In order to further improve the performance of BIO, the following methods are proposed.
    [004] In a conventional bi-prediction in HEVC, the predictor is generated using equation (1), where P ( °) and P (1 > are
    Petition 870190090540, of 12/09/2019, p. 41/91
    3/35 the listO and listi predictor, respectively.
    PConventional ^, j] = | P ^ [z, j] + P ^ [Í, j] + 1 | »1 '(1) [005] In JCTVC-C204 and VECG-AZ05, the BIO predictor is generated using equation (2).
    POplicalFlow = (P W [í, j] + | f, j] + V, [/, - / ,. [/, /]) + v, [z, j] (/, (0) - /, ( 1) [z, j]) + 1) »1 ^ / [006] In equation (2), Ιχ <0 > and Ix (1) represent the directional gradient x in the listO and listi predictor, respectively; I y <0 > and Iy (1) represent the gradient of directional y in listO and listi predictor, respectively; Vx and vy represent the deviations or displacements in the x and y directions, respectively. The derivation process of v x and v y is shown below. First, the cost function is defined as diffCost (x, y) to find the best values v x and v y . To find the best values v x and v y to minimize the cost function, diffCost (x, y), a 5x5 window is used. The solutions of v x and v y can be represented using Si, S2, S3, S5, and Sg. diffCost (x, y)
    X p ° í λ. dP 0 (x, y) x dP x, y) dP x, y) 2 (3) = ZJP + ½ T --- + Vj - (P (py) -Vx --Vy)), q ox uy oxoy = y (P »+ yhi + MAL) + „ (* W + MAjy.
    Ω dx dx dydy [007] The minimum cost function, mindiffCosi (x, y) can be derived according to:
    ddiffCost (x, y) = θ ddiffCost (x, y) = θ dvxdvy [008] Solving the equations (3) and (4), v x and v y can be solved in accordance with equation (5): S3S5-S2S6 SiSè - S3S2
    Vx = --------------, Vy = (5)
    SlS5 - S2S2 SlS5 - S2S2 where,
    Petition 870190090540, of 12/09/2019, p. 42/91
    4/35 ç ν ζθΡ (x, y) dP (x, y). 2 s, = P ^^ + ^^ ) Χ! φ ax dP ° (x, y), dP x, y) dP ° (x, y) dP l (x, y) + dy dx dy Xj = _y ( M®® + MW) ) ( p. (x , y) _p. u , y)) i Ss = y { ^ V £ L + ^ V £ Lya dx dx a dy dy S6 = _ S ( a ^ + M ^ ) (P. Uv) _ plUv)) ω dy oy dP ° (x, y) [009] In the above equations, ------ corresponds to the dx gradient of x of a pixel in (x, y) dP l (x, y)
    Irsta 0, ------ is the qradrente of
    3x a pixel in (x, y) in the list image dy corresponds to the gradient of y direction of a pixel in (x, y) in the image of direction x of dP ° (x, y) in the list image 0, and direction y of a pixel in dP 1 (x, y)
    ------ corresponds to the gradient of dy (x, y) in list 1 image.
    [0010] In some related techniques, S2 can be ignored and v x and v y can be solved according to
    S3 Sè - VxS2
    Vx = ---, Vy = ----------- (6)
    S5 where, dP ° (x, y) dx dP l (x, y) dx ϊ == Σ <
    Ω dP ° (x, y) dx dP l (x, y) dP ° (x, y) dP l (x, y dx dy dy ç v 3P ° ( x , y) 3P '( x , y) ο λ PÚ v S3 = -> (------- + —----) (P (x, y) -p (xy)), the oxox dP ° (x, y) dy dP l (x , y) 2 dy ç Ví 3P ° (x, y), dp l (x, y) o
    5β = -> (- --- + - --- P (x, y) -P (xy)) ω dyoy [0011] According to the above equations, division operations are necessary in the derivation of v x ev y . The required bit depth is large since there are many accumulations and square of pixel values in Si, S2, S3, S5, and Se. If the pixel depth bit depth in the video streams is 10 bits, you need a divider that supports 32 bits / 28 bits for v x and another divider that supports 36 bits / 29 bits for v y . If the bit depth is increased by the fractional interpolation filter,
    Petition 870190090540, of 12/09/2019, p. 43/91
    5/35 then the divisors will become 40 bits / 36 bits for v x and 44 bits / 37 bits for vy. It is impractical to directly use a lookup table (LUT) to replace the split when the required bit depth becomes so large. Therefore, it is desirable to develop methods to simplify division operations in the BIO process.
    [0012] In the equations above, parameters Si, S2, S3, Ss, and Se are related to the gradient of direction x and the gradient of direction y. For example, Si is calculated from the sum of the x direction gradient of the reference block in list 0 and the x direction gradient of the reference block in list 1. The sum square is used as
    51. Ss is calculated from the sum of the y direction gradient of the reference block in list 0 and the y direction gradient of the reference block in list 1. The sum square is used as S 5. For convenience, Si parameters , S2, S3, Ss, and Se are referred to as gradient parameters in this disclosure. In practice, the gradient parameters Si,
    52, S3, Ss, and Se often use fixed point representation with a predefined bit depth. The derivation of v x and v y will require multiplication, addition and division operations. Among them, the division's operation is more expensive to implement.
    [0013] In VCEG-AZ05, the BIO is implemented over the HEVC reference software and is always applied to the blocks provided in true bi-directions (that is, the true bidirectional predicted blocks). In HEVC, an 8-lead interpellation filter for the luma component and a 4-lead interpellation filter for the chroma component are used to perform motion compensation.
    Petition 870190090540, of 12/09/2019, p. 44/91
    Fractional 6/35. Considering a 5x5 window for a pixel to be processed in an 8x8 encoding unit (CU) in BIO, the bandwidth required in the worst case is increased from (8 + 7) x (8 + 7) x 2 / (8x8) = 7.03 for (8 + 7 + 4) x (8 + 7 + 4) x 2 / (8x8) = 11.28 reference pixels per current pixel. In JVET D0042 (A. Alshin, et al., AHG6: In BIO memory bandwidth, ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG Joint Video Exploration Team (JVET) 11, Fourth meeting: Chengdu, CN, October 15-21, 2016, Document: JVET-D0042), to reduce the required bandwidth, the necessary data including motion compensated pixels and gradients in the x and y directions will be set to zeroes, if these pixels are outside the current tile. As shown in Figure 2, the central square is the original MC region (210) for the current block, and the BIO process requires the predictors and the corresponding gradient in Region A (220) and Region B (230) to derive the offset respectively in order to further refine predictors using derived displacement and gradients. In JVET D0042, data from Region A (220) and Region B (230) will be set to zero to save the required bandwidth. As the gradient is generated from additional 6-lead filters, pixels compensated for motion and gradient can be generated using the same region as in the original HEVC. Therefore, using the method in JVET D0042, there is no additional bandwidth requirement in the BIO process.
    [0014] In D0042, the coefficient jump in Region A (220) and Region B (230) is applied to each sub-block if the sub-block's motion compensation is applied. At the
    Petition 870190090540, of 12/09/2019, p. 45/91
    However, if two adjacent sub-blocks have the same MV, these two sub-blocks cannot be merged as one larger block for the BIO process because the final results of two separate sub-blocks and the results of a merged block are different due to the coefficient jump. For example, the coefficient jump can be applied to the internal contour between these two sub-blocks in the separate sub-block process, while the coefficient jump is not applied to the internal contour in the merged block process.
    [0015] However, the use of additional 6-lead filters to generate gradients in the x and y directions is complicated compared to the original motion compensation design. Two additional 6-lead filters will be required: one is used to generate gradients and the other is used to perform interpellation when the motion vector is fractional. In addition, the block-based process is also proposed to reduce the computational complexity required in VCEG-AZ05. Therefore, a low complexity BIO is presented below. In low complexity BIO, the block-based derivation process is used instead of the pixel-based derivation process, where each block is divided into 4x4 blocks (referred to as BIO blocks in this disclosure) for the block-based BIO process. In the original BIO, a 5x5 window is used to derive motion for each pixel. In block-based BIO, a 6x6 window is used for each 4x4 block to derive motion for the block. Gradient calculation is also simplified by applying a 3-lead filter with coefficients equal to {-1, 0, 1} in motion compensated pixels according to the low complexity BIO. At
    Petition 870190090540, of 12/09/2019, p. 46/91
    8/35
    Figure 2, the smaller block 250 corresponds to a 4x4 block for block-based BIO. The block of dashed lines 260 corresponds to the 6x6 window to derive the movement information associated with the block 4x4. For each pixel within the window, the predictor and gradient must be calculated. For the {-1, 0, 1} gradient filter, the gradient in the x and y directions for pixel 240 in the upper left corner of window 260 needs adjacent pixels shown as black dots in Figure 2. Therefore, the width of required bandwidth is the same as in the original BIO, but no additional 6-lead filter is required and computation complexity is reduced using the block-based lead process. It is desirable to develop methods for
    reduce bandwidth in required memory and improve performance even more in low BIO coding complexity. [0016] In one system in typical video encoding that uses inter prediction compensated for movement, the
    Motion information is normally transmitted from an encoder side to a decoder, so that the decoder can correctly perform intercompensated motion prediction. In such systems, the movement information will consume some encoded bits. In order to improve the coding efficiency, a method of derivation of motion vector on the decoder side is revealed in VCEG-AZ07 (Jianle Chen, et al., Additional improvements to HMKTA-1.0, ITU - Telecommunications Standardization Sector, Study Group 16 Question 6, Expert Group on Video Coding (VCEG), 52nd Meeting: 19-26 June 2015, Warsaw, Poland). In
    Petition 870190090540, of 12/09/2019, p. 47/91
    9/35 According to VCEG-AZ07, ο decoder-side motion vector derivation method uses two upward frame rate conversion (FRUC) modes. One of the FRUC modes is referred to as bilateral match for B-slice and the other of the FRUC modes is referred to as model match for P-slice or B-slice.
    Superimposed block movement compensation (OBMC) [0017] Superimposed block movement compensation (OBMC) is a motion compensation technique that estimates the intensity value of a pixel based on motion compensation signals derived from its motion vectors. block movement (MVs) close by, according to the criterion Average Minimum Linear Square Error (LMMSE). From a theoretical-estimation perspective, these VMs are considered plausible hypotheses for their true movement, and for maximizing coding efficiency. Their weights should minimize the mean square forecast error, subject to the unit gain constraint.
    [0018] When High Efficiency Video Coding (HEVC) was developed, several OBMC-based coding techniques were proposed to provide encoding gain. Some of them are described below.
    [0019] In JCTVC-C251, OBMC was applied to the geometric partition. In the geometric partition, it is very likely that a transformation block contains pixels belonging to different partitions. In the geometric partition, since two different motion vectors are used for motion compensation, the pixels in the partition outline can have large discontinuities that can produce block-like visual artifacts. This, in turn,
    Petition 870190090540, of 12/09/2019, p. 48/91
    10/35 will decrease the transformation coding efficiency. Leave the two regions created by a geometric partition denoted by region 1 and region 2. A pixel from region 1 (2) is defined as a boundary pixel if any of its four connected neighbors (ie left, top, right and bottom ) belong to region 2 (1). Figure 3 shows an example of region partition, in which a block 310 is partitioned into region 1 and region 2 through region contour 320. Pixel 330 indicates a region 1 contour pixel and pixel 340 indicates a contour pixel of region 2. If a pixel is a contour pixel, motion compensation is performed using a weighted sum of the movement predictions of the two motion vectors. The weights are 3/4 for the forecast using the motion vector of the region containing the contour pixel and 1/4 for the forecast using the motion vector of the other region. The overlapping contours improve the visual quality of the reconstructed video, in addition to providing coding efficiency in terms of BD rate gain.
    [0020] In JCTVC-F299, OBMC is applied to symmetric motion partitions. If a coding unit (CU) is partitioned into two 2NxN or Nx2N forecast units (PUs), the OBMC will be applied to the horizontal outline of the two 2NxN forecast blocks, and to the vertical outline of the two Nx2N forecast blocks. Since these partitions can have different motion vectors, the pixels in the partition boundaries can have large discontinuities, which can cause visual artifacts and also reduce the efficiency of the transformation / encoding. In JCTVC-F299, OBMC is introduced to smooth out the partition contours of
    Petition 870190090540, of 12/09/2019, p. 49/91
    11/35 movement.
    [0021] Figure 4A illustrates an example of OBMC for partition 2NxN and Figure 4B illustrates an example of OBMC for partition Nx2N. The pixels filled by line represent pixels belonging to Partition 0 and the blank pixels represent pixels belonging to Partition 1. The region superimposed on the luma component is defined as two lines or two columns of pixels on each side of the horizontal or vertical outline, respectively. For rows or columns of pixels (that is, pixels labeled A in Figure 4A and Figure 4B) that are adjacent to the partition outline, the OBMC weighting factors are (3/4, 1/4). For pixels (that is, pixels labeled B in Figure 4A and Figure 4B) that are two rows or columns from the partition outline, the OBMC weighting factors are (7/8, 1/8). For chroma components, the overlapping region is defined as a line or column of pixels on each side of the horizontal or vertical outline, respectively, and the weighting factors are (3/4, 1/4).
    [0022] Currently, the MC results for the overlapping region between two CUs or PUs are generated by another MC process after the normal MC process for the current CU or PU. Therefore, BIO is applied twice in these two MC processes to refine these two MC results, respectively. When two neighboring VMs are the same, the processing order above can help skip the redundant OBMC and BIO processes. However, the required bandwidth and MC operations for the overlapping region are increased compared to the integrated OBMC process and the normal MC process. For example, the size
    Petition 870190090540, of 12/09/2019, p. 50/91
    Current 12/35 of the PU is 16x8, the overlapping region is 16x2 and the interpellation filter in MC is 8 leads. If the OBMC is performed after the normal MC, (16 + 7) x (8 + 7) + (16 + 7) x (2 + 7) = 552 reference pixels per reference list for the current PU and the Related OBMC will be required. If OBMC operations are integrated with the normal MC, only (16 + 7) x (8 + 2 + 7) = 391 reference pixels per reference list are required for the current PU and the related OBMC. Therefore, it is desirable to reduce the computational complexity or memory bandwidth of the BIO when the BIO and OBMC are activated simultaneously.
    BRIEF SUMMARY OF THE INVENTION [0023] Method and apparatus for using bi-directional optical flow (BIO) for a true bi-directional predicted block are disclosed. According to a method of the present invention, input data associated with a current block in a current image is received, in which the current block is encoded using bidirectional prediction. A first reference block associated with the current block in a first reference image from list 0 of the reference image and a second reference block associated with the current block in a second reference image from list 1 of reference images are determined, where the first reference block and the second reference block are indicated by true bidirectional motion vectors. The first x gradients and first y gradients are derived based on the first reference block, and the second x gradients and second y gradients are derived based on the second reference block. A set of parameters
    Petition 870190090540, of 12/09/2019, p. 51/91
    13/35 gradient is derived based on the first gradients x, the first gradients y, the second gradients x, the second gradients y, the first pixel values of the first reference block, the second pixel values of the second reference block or in a combination thereof, where all gradient parameters use fixed point representations. The movement displacement x and the movement displacement y are derived using operations including displacement to the right without any division operation, where the movement displacement x corresponds to a first numerator divided by a first denominator, and the movement displacement y corresponds to a second numerator divided by a second denominator, and where the first numerator, the first denominator, the second numerator and the second denominator are generated from linear combinations of products of two gradient parameters or linear combinations of gradient parameters. A refined predictor is generated for the current block according to the first reference block, the second reference block, the first x gradients and the first y gradients, the second x gradients and the second y gradients, the movement offset x, the offset of movement y, or a combination of them. The current block is encoded or decoded based on the refined predictor.
    [0024] The method may further comprise determining a first MSB position, Nx, of the first denominator and a second MSB position, Ny, of the second denominator, in which the movement displacement x is derived using operations including displacement to the right of the first numerator and of
    Petition 870190090540, of 12/09/2019, p. 52/91
    14/35 first denominator by Nx bits and right shift of the second numerator and the second denominator by Ny bits. The values of (2Px / 2 (Kl)) can be stored in a first look-up table and the values of (2Py / 2 (Kl)) can be stored in a second look-up table, where 2 (K-1) corresponds to possible values of the first denominator and the second denominator, and the movement displacement x is derived using operations that include multiplication and displacement to the right by Px bits and the movement displacement y is derived using operations that include multiplication and displacement to the right by Py bits. In one embodiment, the first lookup table is the same as the second lookup table. In another embodiment, the first lookup table is different from the second lookup table. In one embodiment, K corresponds to 1. The right shift of the first numerator and the first denominator by Nx bits or the right shift of the second numerator and the second denominator by Ny bits can use a rounding offset.
    [0025] A non-transitory, computer-readable medium that stores program instructions that causes a device processing circuit to perform the video encoding method above is also revealed. As mentioned above, the method comprises receiving input data; determine the first reference block and the second reference block; derive the first gradients x and first gradients y based on the first reference block, and the second gradients x and second gradients y based on the second reference block; derive a set of gradient parameters; derive the
    Petition 870190090540, of 12/09/2019, p. 53/91
    15/35 movement displacement x and movement displacement y; generate a refined predictor; and encode or decode the current block based on the refined predictor.
    [0026] According to another method, after the first reference block and the second reference block are determined, a refined predictor is generated for the current block applying the BIO process to the first reference block and the second reference block, wherein said application of the BIO process comprises applying a BIO process dependent on boundary condition conditionally on boundary pixels associated with the first reference block and the second reference block depending on a subblock or block boundary condition, where the boundary condition dependent BIO process comprises filling in the unavailable data needed for a regular BIO process or skipping the unavailable data by modifying the regular BIO process. The unavailable data corresponds to the required destination data for the regular BIO process, but is not available in the first reference block and the second reference block. The current block is encoded or decoded based on the refined predictor.
    [0027] The target data may correspond to the target reference data of the first reference block and the second reference block, target reference data gradients, movement-compensated target reference data, or a combination thereof. In one embodiment, the boundary condition-dependent BIO process is applied to the boundary pixels if the boundary pixels correspond to the block outlines
    Petition 870190090540, of 12/09/2019, p. 54/91
    16/35 of the current block, and the BIO process dependent on boundary condition is not applied to the boundary pixels if the boundary pixels correspond to the block contours compensated by movement of the current block. In another embodiment, the boundary condition dependent BIO process is applied to the boundary pixels if the boundary pixels correspond to a sub-block outline between two adjacent sub-blocks with two different motion vectors, a horizontal sub-block outline between two vertically adjacent sub-blocks or a vertical sub-block outline between two horizontally adjacent sub-blocks. In yet another embodiment, the boundary condition dependent BIO process is not applied to the contour pixels if the contour pixels correspond to a sub-block contour between two adjacent sub-blocks with the same motion vector, a sub-contour -Vertical block between two horizontally adjacent sub-blocks with the same movement vector, or a horizontal sub-block outline between two vertically adjacent sub-blocks with the same movement vector.
    [0028] A non-transitory, computer-readable medium that stores program instructions that causes a device's processing circuit to perform the video encoding method above, is also revealed. As mentioned above, the method comprises receiving input data; determine the first reference block and the second reference block; generate a refined predictor for the current block using the BIO process, in which the application of the BIO process comprises the application of a BIO process dependent on the boundary condition
    Petition 870190090540, of 12/09/2019, p. 55/91
    17/35 conditionally to contour pixels; and encode or decode the current block based on the refined predictor.
    BRIEF DESCRIPTION OF THE DRAWINGS [0029] Figure 1 illustrates an example of Bidirectional Optical Flow (BIO) to derive the displacement motion vector for movement refinement.
    [0030] Figure 2 illustrates the motion compensated region and its neighboring pixels necessary to derive the gradient and displacement related to the Bidirectional Optical Flow (BIO).
    [0031] Figure 3 shows an example of region partition, where a block is partitioned into region 1 and region 2 through the region contour.
    [0032] A Figure4A illustrates one example in OBMC for 2NxN partition.[0033] A Figure 4B illustrates one example in OBMC for Nx2N partition.[0034] A Figure 5A illustrates one example in inheritance from
    offsets, where offsets from a current encoding unit are copied from a neighboring encoding unit.
    [0035] Figure 5B illustrates an example of displacement inheritance, in which the displacements of a current block are copied from neighboring sub-blocks.
    [0036] Figure 6 illustrates an exemplary flowchart of a video encoding system using bidirectional optical flow (BIO) to refine the movement to a predicted bidirectional block according to a modality of the present invention, where displacements are derived using displacement at right without the need for
    Petition 870190090540, of 12/09/2019, p. 56/91
    18/35 division operations.
    [0037] Figure 7 illustrates an exemplary flowchart of a video encoding system using bidirectional optical flow (BIO) to refine the predictor of a predicted bidirectional block according to a modality of the present invention, in which the condition-dependent BIO process contour is applied conditionally to the contour pixels associated with reference blocks k depending on a sub-block or block boundary condition.
    DETAILED DESCRIPTION OF THE INVENTION [0038] The following description is the best way to carry out the invention. This description is made for the purpose of illustrating the general principles of the invention and should not be taken in a limiting sense. The scope of the invention is best determined by reference to the appended claims.
    [0039] As mentioned earlier, the bidirectional optical flow (BIO) and the BIO combined with the Overlapping Block Motion Compensation (OBMC) require access to additional reference data, which causes an increase in system bandwidth. These processes also increase computational complexity. In the present invention, techniques for reducing system bandwidth and / or computational complexity associated with BIO, and BIO with OBMC are disclosed. BIO process without division [0040] As mentioned earlier, the BIO process requires deriving the displacements v x and vy, as shown in equations (5) and (6). The necessary calculations are quite extensive. In particular, division operations are
    Petition 870190090540, of 12/09/2019, p. 57/91
    19/35 necessary to derive the displacements v x and v y based on the gradient parameters. As is known in the art, the computational complexity involved with the division operation is much greater than the addition and multiplication operations. Therefore, it is desirable to avoid split operations.
    [0041] Therefore, a method is revealed to use dynamic right shift to reduce the bit depth required in the splitting process or to eliminate the splitting process. The proposed method is the first to determine the denominator's MSB position and only to maintain the first K denominator MSB bits using right shift by N-bit operation. At the same time, the numerator is also shifted to the right by N bits. The possible values of the denominator become 2 (K-1) and the value of (2P / 2 (K-1)) can be pre-calculated and stored in a lookup table (LUT). Thus, the division operation can be replaced using multiplication and right shift by P bits, as shown in the following equations:
    Vx = ((S 3 »Nx) x LUT [(Si» Nx)]) »Px, ev y = ((S 6 -VxS 2 )» Ny) x LUT [(S 5 »Ny)])» Py.
    [0042] In the above equations, Nx and Ny represent the number of bits associated with the right shift in the denominators, Si and Ss, respectively. Px and Py represent the number of bits associated with the shift to the right of the LUT for v x and v y , respectively. The LUTs for Vx and v y may be the same or different. In one embodiment, a rounding offset can be added when performing the right shift operation. In another
    Petition 870190090540, of 12/09/2019, p. 58/91
    20/35 mode, only the first MSB bit of the denominator is maintained. In this case, K corresponds to m and only the dynamic shift to the right is necessary to replace the division in the BIO process.
    Displacement reuse v x ev y in the OBMC region [0043] In the BIO process, small displacements (also referred to as deviations in this disclosure), v x and ev y , are derived based on the pixel value gradients in the x and y directions used to refine the predictor of MC. This derivation can be block-based or pixel-based operations. To reduce the computational complexity of the BIO-related OBMC process, small displacements in the normal MC process are reused for the blocks or pixels in the OBMC region according to this method. In other words, in the BIO-related OBMC process, small displacements can be copied or inherited from those in the normal MC process. For example, small offsets of blocks or contour pixels in the normal MC region can be used directly for those pixels or blocks in the related OBMC region. Figure 5A illustrates an example of offset inheritance, where offsets from a current coding unit can be used for a neighboring coding unit. For example, offsets from the current coding unit 510 can be used for neighboring block 512; the offsets of the current coding unit 520 can be used directly for the neighboring coding unit 522 or 524. Figure 5B illustrates an example of offset inheritance, where the offsets of a current block 530 can be used for neighboring sub-blocks. For example,
    Petition 870190090540, of 12/09/2019, p. 59/91
    21/35 offsets from the current block 530 can be used for neighboring sub-block 532, 534, 536 or 538.
    Processing order for OBMC and BIO [0044] Currently, the processing order for OBMC and BIO is to apply BIO first to refine MC predictors, and then apply OBMC to mix two refined MC predictors. The processing order is changed to reduce computational complexity according to this method. According to this method, the OBMC mixture is performed first to generate a new MC predictor and the BIO process is applied later. Therefore, the BIO process can be reduced from twice to once.
    Bandwidth reduction for BIO [0045] In a low complexity BIO process, a 3-lead filter with coefficients equal to [-1, 0, 1] was used in the motion compensated pixels. In the low complexity BIO, the motion-compensated pixels of Region A (220) and Region B (230) in Figure 2 are used to calculate the gradients, and these gradients are used to derive the offset and refine the final predictor. As shown in Figure 2, motion-compensated pixels from Region A (220) and Region B (230) are outside the contour of the motion-compensated block 210. To reduce bandwidth, calculating the gradient for pixels in the CU contour is modified to avoid the need for data from Region A (220) and Region B (230). For example, a 2-lead filter with filter coefficients equal to [-1, 1] is used to generate gradients for these pixels in the CU contours. Since the gradient calculation is different,
    Petition 870190090540, of 12/09/2019, p. 60/91
    22/35 some normalization may be required based on the filter bypass distance. In another modality, the BIO refinement is skipped for these pixels in the CU contours.
    Block size selection for BIO process [0046] In low complexity BIO, the block-based derivation process is used instead of the pixel-based derivation process, which can significantly save computational complexity. However, using a predefined block size may not achieve the best encoding gain for videos with different resolutions and content. According to the present method, the block size may depend on the resolution of the video or on the characteristics of the video content. For example, for some smooth regions, the block size must be larger enough to include more textures to derive an accurate offset. However, for some complicated textures or motion regions, the block size must be small to adapt to the location. Therefore, in one mode, the block size depends on the video resolution. In another embodiment, the block size is dependent on the CU size. For example, when the CU size is small, a small block size is used. In another embodiment, the block size depends on the use of the sub-block movement partition. If the current CU is encoded using the sub-block motion partition, a smaller block size will be used for the BIO. In another embodiment, the block size depends on the pixels compensated for movement. For example, if the variation in pixels compensated for movement is large or if there is
    Petition 870190090540, of 12/09/2019, p. 61/91
    23/35 some strong edges in the motion compensated (MC) regions, a smaller block size will be used for the BIO. The above methods can be combined to implicitly determine the block size in the BIO process or explicitly flag the block size selected in the bit stream at the sequence level, image level, slice level, CTU level, or CU level.
    Boundary condition dependent process [0047] To resolve the issue of different results of processing the sub-blocks together or separately, a boundary condition dependent BIO process is revealed. The current BIO boundary process depends on the block / subblock boundary condition. The present BIO contour process refers to the BIO process applied to the contour region. As mentioned earlier, the BIO process requires pixels from outside the current block, as shown in Figure 2. For motion compensation, reference data outside the current block will be needed to support fractional motion vectors. As shown in Figure 2, the region compensated by movement 210 is necessary to support fractional motion vectors. The BIO process may require additional reference data outside the motion compensated region to perform filtering and derive gradients. Depending on whether the processing of the BIO sub-blocks is carried out together or separately, the required data regions may be different. In order to solve the different data requirements, a present invention describes the BIO process dependent on boundary condition. When the necessary data (referred to as region
    Petition 870190090540, of 12/09/2019, p. 62/91
    24/35 required) for the BIO process are not available, the boundary condition dependent BIO process can fill or ignore the unavailable data. The data can correspond to reference pixels, gradients or results compensated by movement or a combination of them. When the necessary data is not available, the unavailable data can be filled. The data filling is known in the state of the art of video and image processing. For example, unavailable data can be filled by repeating outline pixels, repeating mirrored outline pixels, or using predefined pixel values. Unavailable data can also be resolved by ignoring unavailable data. For example, the BIO process in the contour can be modified instead of changing the data source by filling in when the data needed in the BIO process is not in an available region. Consequently, the necessary data is ignored to avoid the need for unavailable data. For example, if reference pixels outside the region available in the BIO interpellation process are needed, a modality of the current method can change the interpellation process (for example, shrink the bypass filter, skip an interpellation filter coefficient, etc. .) to avoid using these reference pixels. In another example, if motion-compensated pixels are needed outside the region available in the BIO gradient calculation, a modality of the current method can change the gradient process (for example, modify the gradient calculation, set the gradients to 0, etc.) to avoid the need for
    Petition 870190090540, of 12/09/2019, p. 63/91
    25/35 generate these motion compensated pixels. In yet another example, if gradients outside the available region are needed in the BIO derivation process, a modality of the current method can alter the derivation process (for example, decrease the window size, skip the gradients, etc.) to avoid the need to generate these gradients.
    [0048] In a modality, if the contour is a CU / PU contour, the reference pixel filling, the gradient or the compensation result is applied, or the coefficient jump, reference pixel, gradient or compensation result. If the contour is a sub-block contour and the adjacent MVs are the same, the fill of the reference pixel, gradient or compensation result, or jump of the coefficient, reference pixel, gradient or compensation result is not applied. If the contour is a sub-block contour and the adjacent MVs are different, fill in the reference pixel, result of gradient compensation, or jump of the coefficient, reference pixel, result of gradient or compensation is applied. Consequently, the results of processing the sub-blocks together or separately will be the same.
    [0049] In another mode, if the contour is the CU / PU contour, fill in the reference pixel, result of gradient or compensation, jump of the coefficient, reference pixel, result of gradient or compensation is applied to save the width of memory band. If the contour is the vertical sub-block contour (that is, the contour between two adjacent horizontal sub-blocks) and
    Petition 870190090540, of 12/09/2019, p. 64/91
    26/35 the adjacent MVs are the same, fill in the reference pixel, result of gradient or compensation, or skip the coefficient, reference pixel, result of gradient or compensation is not applied. If the contour is the sub-block contour and the adjacent MVs are different or horizontal sub-block contours, filling in the reference pixel, gradient or compensation result, or jump of the coefficient result, reference pixel, result of gradient or compensation. Consequently, the results of processing the sub-blocks together or separately in the horizontal direction will be the same.
    [0050] In another modality, if the contour is the CU / PU contour, fill in the reference pixel, result of gradient or compensation, jump of the coefficient, reference pixel, result of gradient or compensation is applied to save the width of memory band. If the contour is the horizontal contour of the sub-block (the contour between two adjacent vertical sub-blocks) and the adjacent MVs are the same, fill in the reference pixel, gradient or compensation result, or jump of the coefficient, reference pixel , gradient or compensation result is not applied. If the contour is the sub-block contour and the adjacent MVs are different contours or vertical sub-blocks, fill in the reference pixel, gradient or compensation result, or jump of the coefficient, reference pixel, gradient or compensation result is applied. Therefore, the results of processing the sub-blocks together or separately in the vertical direction will be the same.
    Petition 870190090540, of 12/09/2019, p. 65/91
    27/35 [0051] In yet another modality, for each block contour, regardless of the PU / CU contour or sub-block contour, filling of the reference pixel, gradient or compensation result, or jump of the coefficient, pixel of reference, gradient or compensation result is not applied if adjacent two-sided VMs are the same. Otherwise, the fill in of the reference pixel, gradient or compensation result, or jump of the coefficient, reference pixel, gradient or compensation result is applied.
    [0052] In yet another modality, for each block contour, except the CTU (coding tree unit) or CTU line contour, if the adjacent two-sided VMs are the same, fill in the reference pixel, gradient or result compensation, or coefficient jump, reference pixel, gradient, or compensation result is not applied. Otherwise, the fill of the reference pixel, gradient or compensation result is applied, or the jump of the coefficient, reference pixel, gradient or compensation result.
    OBMC and BIO [0053] Overlapping block motion compensation is generally used to reduce block artifact in video encoding. In JEM, for a PU / CU, the upper rows and columns on the left are weighted using the upper block MVs and the left block MVs. In other words, the current block's motion compensation will be extended further to the right and bottom to compensate for the right neighboring block and the lower neighboring block.
    Petition 870190090540, of 12/09/2019, p. 66/91
    28/35
    [0054] How O OBMC can be combined in the process normal MC, at BIO operations are integrated with those of normal process in MC, according with the present method.
    Therefore, some data required by the BIO process can reuse the data used in the normal MC process, which includes the MC predictor, pixel gradients and intermediate results of MC interpellation to avoid redundant computation and reduce memory bandwidth. needed. For example, the PU size can be MxN and the related OBMC size can be Mx4. If the MC with BIO for the current PU and the related OBMC are run separately, (M + 4 + 7) x (N + 4 + 7) and (M + 4 + 7) x (4 + 4 + 7) pixels of reference by reference list will be required, respectively. If the OBMC process integrated with the normal MC process is used, only (M + 4 + 7) x (N + 4 + 4 + 7) reference pixels per reference list are needed. If the PU size is 16x8, the memory bandwidth can be reduced from 918 reference pixels per reference list to 621 reference pixels per reference list. If the OBMC process is applied to more than one contour, the reduction in memory bandwidth will become greater. Some intermediate data generated in the normal MC process with BIO can be reused in the related OBMC process. For example, the intermediate results can be partial results in the interpellation process or the gradient values used in the BIO. Therefore, the reuse of this intermediate data only reduces the memory bandwidth, but it also reduces the necessary computational complexity of the related OBMC process.
    Petition 870190090540, of 12/09/2019, p. 67/91
    29/35 [0055] In another modality, the BIO is applied when the OBMC can be combined in the normal MC process. The OBMC region can include the right region and the lower region, or even the upper region and the left region if the MC sub-block is applied. Since the OBMC will load additional reference pixels for the right region and the lower region, the BIO of the current block does not need to perform the result fill for the reference pixel, gradient or offset, or the result jump for the coefficient, pixel of reference, gradient or compensation because the data is already loaded by the OBMC. According to a modality, the result fill for the reference pixel, gradient or compensation, or the result jump for the coefficient, reference pixel, gradient or compensation, or any contour dependent process to reduce the bandwidth in the OBMC region contour is applied. These techniques will not be applied to the contour between the current block and the OBMC regions.
    [0056] In another mode, BIO is still disabled in the OBMC region. That is, the BIO is applied only to the current block and the contour-dependent process to reduce the memory bandwidth is not applied, if the necessary data is loaded by the OBMC region.
    [0057] The boundary condition dependent BIO process and integrated OBMC and BIO can be combined. For example, to perform ο ΒΙΟ, the contour between the current block and the OBMC block does not need to apply result fill for the reference pixel, gradient or offset, or the result jump for the coefficient, reference pixel, gradient or compensation, or any
    Petition 870190090540, of 12/09/2019, p. 68/91
    30/35 contour dependent process to reduce bandwidth.
    [0058] Figure 6 illustrates an exemplary flowchart of a video encoding system using bidirectional optical flow (BIO) to refine the movement to a predicted bidirectional block according to an embodiment of the present invention, in which displacements are derived using displacement to the right without the need for split operations. The steps shown in the flowchart, as well as other flowcharts in this description, can be implemented as program codes executable on one or more processors (for example, one or more CPUs) on the encoder side and / or on the decoder side. The steps shown in the flowchart can also be implemented on a hardware basis as one or more electronic devices or processors arranged to perform the steps in the flowchart. According to this method, the input data associated with a current block in a current image is received at step 610, where the current block is encoded using bidirectional prediction. A first reference block associated with the current block in a first reference image from list 0 of reference images and a second reference block associated with the current block in a second reference image from list 1 of reference images are derived in step 620 , where the first reference block and the second reference block are indicated by true bidirectional motion vectors. The first x gradients and first y gradients are derived based on the first reference block and the second x gradients and second gradients
    Petition 870190090540, of 12/09/2019, p. 69/91
    31/35 y are derived based on the second reference block in step 630. The definition of x gradients (ie, gradients in the x direction) and y gradients (ie, gradients in the y direction) was revealed earlier in the disclosure. A set of gradient parameters is derived based on the first gradients x, the first gradients y, the second gradients x, the second gradients y, the first pixel values of the first reference block, the second pixel values of the second block of reference, or a combination of them in step 640, and where all gradient parameters use fixed point representations. As mentioned earlier, the gradient parameter set can correspond to the gradient parameters Si, S2, S3, S5 and Sg, as shown in equations (5) and (6). Movement shift x and movement shift y are derived using operations including shift to the right with no division operation in step 650, where movement shift x corresponds to a first numerator divided by a first denominator, and movement shift y corresponds to a second numerator divided by a second denominator, and where the first numerator, the first denominator, the second numerator and the second denominator are generated from linear combinations of products of two gradient parameters or linear combinations of gradient parameters. As shown in equations (5) and (6), the displacements can be derived from the gradient parameters Si, S2, S3, S5, and Sg. However, the present invention uses right shift operations to derive offsets from gradient parameters to
    Petition 870190090540, of 12/09/2019, p. 70/91
    32/35 avoid the need for division operations. A refined predictor for the current block is generated according to the first reference block, the second reference block, the first x gradients and the first y gradients, the second x gradients and the second y gradients, the movement offset x, the offset of motion y or a combination of these in step 660. Equation (2) illustrates an example of a refined predictor derivation using displacements. The current block is encoded or decoded based on the refined predictor in step 670.
    [0059] Figure 7 illustrates an exemplary flowchart of a video encoding system using bidirectional optical flow (BIO) to refine the predictor of a predicted bidirectional block according to a modality of the present invention, in which the condition-dependent BIO process contour is applied conditionally to the contour pixels associated with reference blocks k depending on a block or subblock contour condition. According to this method, the input data associated with a current block in a current image is received at step 710, where the current block is encoded using bidirectional prediction. A first reference block associated with the current block in a first reference image from list 0 of reference images and a second reference block associated with the current block in a second reference image from list 1 of reference images are derived in step 720 , where the first reference block and the second reference block are pointed by bidirectional motion vectors
    Petition 870190090540, of 12/09/2019, p. 71/91
    33/35 true. A refined predictor is generated for the current block by applying the BIO process to the first reference block and the second reference block in step 730, in which said application of the BIO process comprises the application of a BIO process dependent on the boundary condition conditionally to the contour pixels associated with the first reference block and the second reference block depending on a block or sub-block boundary condition, and where the boundary condition-dependent BIO process comprises filling in unavailable data needed for a regular or ignoring BIO process the unavailable data modifying the regular BIO process, and the unavailable data correspond to the required destination data for the regular BIO process, but not available in the first reference block and the second reference block. The current block is encoded or decoded based on the refined predictor in step 740.
    [0060] The flowcharts shown above are intended to illustrate an example of video encoding according to the present invention. A person skilled in the art can modify each step, rearrange the steps, divide a step or combine steps to practice the present invention without departing from the spirit of the present invention. In the disclosure, specific syntax and semantics were used to illustrate examples for implementing modalities of the present invention. One skilled in the art can practice the present invention
    replacing the syntax and The semantics by syntax and equivalent semantics without if to put away of the spirit gives present invention.[0061] The description above is presented to allow
    Petition 870190090540, of 12/09/2019, p. 72/91
    34/35 that a person skilled in the art practices the present invention, as envisaged in the context of a particular application and its requirement. Various modifications to the described modalities will be evident to those skilled in the art, and the general principles defined here can be applied to other modalities. Therefore, the present invention is not intended to be limited to the particular modalities shown and described, but must be in accordance with the broader scope consistent with the principles and innovative features described herein. In the detailed description above, several specific details are illustrated in order to provide a complete understanding of the present invention. However, it will be understood by those skilled in the art that the present invention can be practiced.
    [0062] The mode of the present invention as described above, can be implemented on various hardware, software codes, or a combination of both. For example, one embodiment of the present invention can be one or more circuit circuits integrated into a video compression chip or program code integrated into the video compression software to perform the processing described herein. One embodiment of the present invention can also be a program code to be executed on a digital signal processor (DSP) to perform the processing described herein. The invention can also involve a number of functions to be performed by a computer processor, a digital signal processor, a microprocessor or an FPGA (Field Programmable Port Array). These processors can be configured to perform particular tasks according to
    Petition 870190090540, of 12/09/2019, p. 73/91
    35/35 with the invention, executing machine-readable software code or firmware code that defines the particular methods incorporated by the invention. The software code or firmware code can be developed in different programming languages and different formats or styles. The software code can also be compiled for different target platforms. However, different code formats, styles and software code languages and other means of configuring code to perform the tasks according to the invention will not depart from the spirit and scope of the invention.
    [0063] The invention can be incorporated in other specific forms without departing from its spirit or essential characteristics. The examples described should be considered in all respects only as illustrative and not restrictive. The scope of the invention is therefore indicated by the appended claims and not by the previous description. All changes that fall within the meaning and scope of the claims' equivalence should be covered by their scope.
    权利要求:
    Claims (11)
    [1]
    1. Video encoding method using bidirectional optical flow (BIO) for a true bidirectional predicted block, the method characterized by the fact that it comprises:
    receiving input data associated with a current block in a current image, where the current block is encoded using bidirectional prediction;
    determine a first reference block associated with the current block in a first reference image from list 0 of the reference image and a second reference block associated with the current block in a second reference image from list 1 of reference image, in that the first reference block and the second reference block are indicated by true bidirectional motion vectors;
    derive the first gradients x and first gradients y based on the first reference block, and the second gradients x and second gradients y based on the second reference block;
    derive a set of gradient parameters based on the first x gradients, the first y gradients, the second x gradients, the first pixel values of the first reference block, the second pixel values of the second reference block , or a combination thereof, and where all gradient parameters use fixed point representations;
    derive motion displacement x and motion displacement y using operations including motion displacement
    Petition 870190090540, of 12/09/2019, p. 75/91
    [2]
    2/11 right without any division operation, where the movement displacement x corresponds to a first numerator divided by a first denominator, and the movement displacement y corresponds to a second numerator divided by a second denominator, and where the first numerator, first denominator, second numerator, and second denominator are generated from linear combinations of products of two gradient parameters or linear combinations of gradient parameters;
    generate a refined predictor for the current block according to the first reference block, the second reference block, the first gradients x and the first gradients y, the second gradients x and the second gradients y, the movement offset x, the displacement of y movement, or a combination thereof; and encode or decode the current block based on the refined predictor.
    2. Method, according to claim 1, characterized by the fact that it also comprises determining a first MSB position, Nx, of the first denominator and a second MSB position, Ny, of the second denominator, in which the movement displacement x is derived using operations including right shift of the first numerator and first denominator by Nx bits and right shift of the second numerator and second denominator by Ny bits.
    3. Method, according with the claim 2, characterized by fact that the values of (2 Px / 2 < K -D) are stored in a first table of consultation, and the
    (2 p y / 2 (K D) values are stored in a second table
    Petition 870190090540, of 12/09/2019, p. 76/91
    [3]
    3/11 query, and where 2 (K corresponds to possible values of the first denominator and the second denominator, and the movement offset x is derived using operations including multiplication and right shift by Px bits and the movement offset y is derived using operations including multiplication and right shift by Py bits.
    4. Method, in wake up with the claim 3, characterized by fact of that first table in Query it's the same as second table Query. 5. Method, in wake up with the claim 3, characterized by fact of that first table in Query is different from second lookup table. 6. Method, in wake up with the claim 3, characterized by fact that K corresponds to 1. 7. Method, in wake up with the claim 2, characterized by fact of that the offset to The right of the first numerator < and the first denominator per
    Nx bits or offset to the right of the second numerator and the second denominator per Ny bits uses a rounding offset.
    8. Video encoding device using bidirectional optical flow (BIO) for a true bidirectional predicted block, characterized by the fact that it comprises one or more electronic circuits or processors arranged for:
    receiving input data associated with a current block in a current image, where the current block is encoded using bidirectional prediction;
    determine a first associated reference block
    Petition 870190090540, of 12/09/2019, p. 77/91
    [4]
    4/11 to the current block in a first reference image from list 0 of reference image and a second reference block associated with the current block in a second reference image from list 1 of reference image, where the first block reference point and the second block reference are indicated by true bidirectional motion vectors;
    derive the first gradients x and first gradients y based on the first reference block, and the second gradients x and second gradients y based on the second reference block;
    derive a set of gradient parameters based on the first gradients x, the first gradients y, the second gradients x, the second gradients y, the first pixel values of the first reference block and the second pixel values of the second reference block , and where all gradient parameters use fixed point representations;
    derive motion displacement x and motion displacement y using operations including displacement to the right without any division operation, where motion displacement x corresponds to a first numerator divided by a first denominator, and motion displacement y corresponds to a second numerator divided by a second denominator, and where the first numerator, the first denominator, the second numerator, and the second denominator are generated from linear combinations of products of two gradient parameters or linear combinations of gradient parameters;
    Petition 870190090540, of 12/09/2019, p. 78/91
    [5]
    5/11 generate a refined predictor for the current block according to the first reference block, the second reference block, the first gradients x and the first gradients y, the second gradients x and the second gradients y, the movement offset x, the displacement of movement y, or a combination thereof; and encode or decode the current block based on the refined predictor.
    9. Non-transitory computer-readable medium, characterized by the fact that it stores program instructions causing a device's processing circuit to execute a video encoding method, and the method comprising:
    receiving input data associated with a current block in a current image, where the current block is encoded using bidirectional prediction;
    determine a first reference block associated with the current block in a first reference image from list 0 of the reference image and a second reference block associated with the current block in a second reference image from list 1 of reference image, in that the first reference block and the second reference block are indicated by true bidirectional motion vectors;
    derive the first gradients x and first gradients y based on the first reference block, and the second gradients x and second gradients y based on the second reference block;
    derive a set of gradient parameters based on the first x gradients, the first gradients
    Petition 870190090540, of 12/09/2019, p. 79/91
    [6]
    6/11 y, in the second gradients x, in the second gradients y, in the first pixel values of the first reference block, in the second pixel values of the second reference block, or a combination thereof, and in which all the parameters of gradient use whole representations, and where all gradient parameters use fixed point representations;
    derive motion displacement x and motion displacement y using operations including displacement to the right without any division operation, where motion displacement x corresponds to a first numerator divided by a first denominator, and motion displacement y corresponds to a second numerator divided by a second denominator, and where the first numerator, the first denominator, the second numerator, and the second denominator are generated from linear combinations of products of two gradient parameters or linear combinations of gradient parameters;
    generate a refined predictor for the current block according to the first reference block, the second reference block, the first gradients x and the first gradients y, the second gradients x and the second gradients y, the movement offset x, the displacement of y movement, or a combination thereof; and encode or decode the current block based on the refined predictor.
    10. Video encoding method using bidirectional optical flow (BIO) for a true bidirectional predicted block, the method characterized by the fact
    Petition 870190090540, of 12/09/2019, p. 80/91
    [7]
    7/11 that comprises: receiving input data associated with a current block in a current image, in which the current block is encoded using bidirectional prediction;
    determine a first reference block associated with the current block in a first reference image from list 0 of the reference image and a second reference block associated with the current block in a second reference image from list 1 of reference image, in that the first reference block and the second reference block are indicated by true bidirectional motion vectors;
    generate a refined predictor for the current block by applying the BIO process to the first reference block and the second reference block, in which the application of the BIO process comprises the application of a boundary condition dependent BIO process conditionally to the boundary pixels associated with the the first reference block and the second reference block depending on a block or sub-block boundary condition, and where the boundary condition dependent BIO process comprises filling in the unavailable data required for a regular BIO process or skipping the unavailable data modifying the regular BIO process, and the unavailable data corresponds to the required destination data for the regular BIO process, but not available in the first reference block and the second reference block; and encode or decode the current block based on the refined predictor.
    Petition 870190090540, of 12/09/2019, p. 81/91
    [8]
    11/11
    11. Method according to claim 10, characterized by the fact that the destination data corresponds to the destination reference data of the first reference block and the second reference block,
    gradients From Dice of reference in destination, data in reference in destiny compensated per movement, or an combination From themselves. 12. 1 Method, of a deal with The claim 10,
    characterized by the fact that the boundary condition dependent BIO process is applied to the boundary pixels if the boundary pixels correspond to the contours of the current block, and the boundary condition dependent BIO process is not applied to the boundary pixels if the contour pixels correspond to the contours of the block compensated by movement of the current block.
    13. Method according to claim 10, characterized by the fact that the BIO process dependent on boundary condition is applied to the boundary pixels if the boundary pixels correspond to a subblock contour between two adjacent sub-blocks with two vectors different motion patterns or a horizontal sub-block contour between two vertically adjacent sub-blocks.
    14. Method, according to claim 10, characterized by the fact that the boundary condition dependent BIO process is applied to the boundary pixels if the boundary pixels correspond to a subblock contour between two adjacent sub-blocks with two vectors different motion patterns or a vertical sub-block contour between two horizontally adjacent sub-blocks.
    15. Method according to claim 10,
    Petition 870190090540, of 12/09/2019, p. 82/91
    [9]
    9/11 characterized by the fact that the boundary condition dependent BIO process is not applied to the contour pixels if the contour pixels correspond to a sub-block contour between two adjacent sub-blocks with the same motion vector.
    16. Method according to claim 10, characterized by the fact that the boundary condition-dependent BIO process is not applied to the contour pixels if the contour pixels correspond to a vertical sub-block contour between two sub-blocks horizontally adjacent with the same motion vector.
    17. Method, according to claim 10, characterized by the fact that the BIO process dependent on boundary condition is not applied to the boundary pixels if the boundary pixels correspond to a horizontal sub-block boundary between two sub-blocks vertically adjacent with the same motion vector.
    18. Video encoding device using bidirectional optical flow (BIO) for a true bidirectional predicted block, the video encoding device characterized by the fact that it comprises one or more electronic circuits or processors arranged for:
    receiving input data associated with a current block in a current image, where the current block is encoded using bidirectional prediction;
    determine a first reference block associated with the current block in a first reference image from list 0 of the reference image and a second reference block associated with the current block in a second reference image from list 1 of reference image, in
    Petition 870190090540, of 12/09/2019, p. 83/91
    [10]
    10/11 that the first reference block and the second reference block are pointed out by true bidirectional motion vectors;
    generate a refined predictor for the current block by applying the BIO process to the first reference block and the second reference block, in which the application of the BIO process comprises the application of a boundary condition dependent BIO process conditionally to the boundary pixels associated with the the first reference block and the second reference block depending on a block or sub-block boundary condition, and where the boundary condition dependent BIO process comprises filling in the unavailable data required for a regular BIO process or skipping the unavailable data modifying the regular BIO process, and the unavailable data corresponds to the required destination data for the regular BIO process, but not available in the first reference block and the second reference block; and encode or decode the current block based on the refined predictor.
    19. Non-transitory computer-readable medium, characterized by the fact that it stores program instructions causing a device's processing circuit to execute a video encoding method, and the method comprising:
    receiving input data associated with a current block in a current image, where the current block is encoded using bidirectional prediction;
    determine a first associated reference block
    Petition 870190090540, of 12/09/2019, p. 84/91
    [11]
    11/11 to the current block in a first reference image from list 0 of reference image and a second reference block associated with the current block in a second reference image from list 1 of reference image, where the first block reference point and the second block reference are indicated by true bidirectional motion vectors;
    generate a refined predictor for the current block by applying the BIO process to the first reference block and the second reference block, in which the application of the BIO process comprises the application of a boundary condition dependent BIO process conditionally to the boundary pixels associated with the the first reference block and the second reference block depending on a block or sub-block boundary condition, and where the boundary condition dependent BIO process comprises filling in the unavailable data required for a regular BIO process or skipping the unavailable data modifying the regular BIO process, and the unavailable data corresponds to the required destination data for the regular BIO process, but not available in the first reference block and the second reference block; and encode or decode the current block based on the refined predictor.
    类似技术:
    公开号 | 公开日 | 专利标题
    BR112019018922A2|2020-04-14|motion refinement method and apparatus based on bi-directional optical flow for video encoding
    BR112019012582A2|2019-11-19|video coding motion refinement method and apparatus
    US10499053B2|2019-12-03|Method of improved directional intra prediction for video coding
    US10397588B2|2019-08-27|Method and apparatus for resource sharing between intra block copy mode and inter prediction mode in video coding systems
    WO2015124110A1|2015-08-27|Method of video coding using prediction based on intra picture block copy
    BR122020002123A2|2020-12-01|VIDEO DECODING
    US20160080766A1|2016-03-17|Encoding system using motion estimation and encoding method using motion estimation
    WO2016138854A1|2016-09-09|Method and apparatus for intrabc mode with fractional-pel block vector resolution in video coding
    WO2019210829A1|2019-11-07|Signaling for illumination compensation
    CN113853794A|2021-12-28|Mutually exclusive settings for multiple tools
    BR112020020976A2|2021-01-19|IMPLIED TRANSFORM ADJUSTMENTS
    BR112021000698A2|2021-04-13|MIXING CANDIDATES WITH MULTIPLE HYPOTHESES
    US20200195925A1|2020-06-18|Refinement of internal sub-blocks of a coding unit
    CN113728639A|2021-11-30|Method and apparatus for optical flow prediction refinement
    KR102327942B1|2021-11-17|Method and device for determining coding unit partitioning, computing device and readable storage medium
    BR112020003938A2|2020-09-08|motion compensation at a finer precision than the motion vector differential
    US20130114721A1|2013-05-09|Predicted motion vectors
    TWI738292B|2021-09-01|Method and apparatus of simplified affine subblock process for video coding system
    WO2020227892A1|2020-11-19|Non-local intra block copy and parallel coding for motion prediction
    CN113994692A|2022-01-28|Method and apparatus for predictive refinement with optical flow
    CN112055207A|2020-12-08|Time domain motion vector prediction method, apparatus and storage medium
    CN113728646A|2021-11-30|Method and apparatus for predictive refinement with optical flow
    同族专利:
    公开号 | 公开日
    TWI663872B|2019-06-21|
    EP3586513A4|2020-12-09|
    EP3586513A1|2020-01-01|
    US11109062B2|2021-08-31|
    CN110476424B|2022-03-04|
    US20210127133A1|2021-04-29|
    CN110476424A|2019-11-19|
    TW201840190A|2018-11-01|
    WO2018166357A1|2018-09-20|
    引用文献:
    公开号 | 申请日 | 公开日 | 申请人 | 专利标题
    EP2557795A4|2010-04-06|2015-07-08|Samsung Electronics Co Ltd|Method and apparatus for video encoding and method and apparatus for video decoding|
    CN103618904B|2013-11-20|2017-02-22|华为技术有限公司|Motion estimation method and device based on pixels|
    EP3332551A4|2015-09-02|2019-01-16|MediaTek Inc.|Method and apparatus of motion compensation for video coding based on bi prediction optical flow techniques|EP3804324A1|2018-06-11|2021-04-14|Mediatek Inc.|Method and apparatus of bi-directional optical flow for video coding|
    KR20210071958A|2018-09-21|2021-06-16|브이아이디 스케일, 인크.|Reduce complexity and control bit width for bidirectional optical flow|
    EP3878178A1|2018-11-05|2021-09-15|InterDigital VC Holdings, Inc.|Video encoding or decoding using block extension for overlapped block motion compensation|
    CN112219400A|2018-11-06|2021-01-12|北京字节跳动网络技术有限公司|Location dependent storage of motion information|
    CN113170097A|2018-11-20|2021-07-23|北京字节跳动网络技术有限公司|Coding and decoding of video coding and decoding modes|
    CN113196777A|2018-12-17|2021-07-30|北京字节跳动网络技术有限公司|Reference pixel filling for motion compensation|
    WO2020177665A1|2019-03-05|2020-09-10|Mediatek Inc.|Methods and apparatuses of video processing for bi-directional prediction with motion refinement in video coding systems|
    CA3131311A1|2019-03-08|2020-09-17|Huawei Technologies Co., Ltd.|An encoder, a decoder and corresponding methods for inter prediction|
    CN113597769A|2019-03-19|2021-11-02|华为技术有限公司|Video inter-frame prediction based on optical flow|
    CN113615197A|2019-03-26|2021-11-05|北京达佳互联信息技术有限公司|Method and apparatus for bit depth control of bi-directional optical flow|
    WO2020200159A1|2019-03-29|2020-10-08|Beijing Bytedance Network Technology Co., Ltd.|Interactions between adaptive loop filtering and other coding tools|
    CN113661708A|2019-04-02|2021-11-16|北京字节跳动网络技术有限公司|Video encoding and decoding based on bidirectional optical flow|
    EP3922015A1|2019-04-19|2021-12-15|Beijing Bytedance Network Technology Co. Ltd.|Gradient calculation in different motion vector refinements|
    CN110225339A|2019-05-10|2019-09-10|上海德衡数据科技有限公司|A kind of HEVC video sequence coding/decoding accelerated method|
    US11089328B2|2019-05-22|2021-08-10|Qualcomm Incorporated|Bi-directional optical flow in video coding|
    WO2021054886A1|2019-09-20|2021-03-25|Telefonaktiebolaget Lm Ericsson |Methods of video encoding and/or decoding with bidirectional optical flow simplification on shift operations and related apparatus|
    CN110798630B|2019-10-30|2020-12-29|北京市商汤科技开发有限公司|Image processing method and device, electronic equipment and storage medium|
    CN111131837A|2019-12-30|2020-05-08|浙江大华技术股份有限公司|Motion compensation correction method, encoding method, encoder, and storage medium|
    WO2021174396A1|2020-03-02|2021-09-10|Oppo广东移动通信有限公司|Image prediction method, encoder, decoder and storage medium|
    WO2021249375A1|2020-06-08|2021-12-16|Beijing Bytedance Network Technology Co., Ltd.|Affine prediction improvements for video coding|
    法律状态:
    2021-10-19| B350| Update of information on the portal [chapter 15.35 patent gazette]|
    优先权:
    申请号 | 申请日 | 专利标题
    US201762472002P| true| 2017-03-16|2017-03-16|
    US201762474702P| true| 2017-03-22|2017-03-22|
    US201762480359P| true| 2017-04-01|2017-04-01|
    PCT/CN2018/077819|WO2018166357A1|2017-03-16|2018-03-02|Method and apparatus of motion refinement based on bi-directional optical flow for video coding|
    [返回顶部]